Papers with long-context degradation
LAMB: A Training-Free Method to Enhance the Long-Context Understanding of SSMs via Attention-Guided Token Filtering (2025.acl-short)
Copied to clipboard
Zhifan Ye, Zheng Wang, Kejing Xia, Jihoon Hong, Leshu Li, Lexington Whalen, Cheng Wan, Yonggan Fu, Yingyan Celine Lin, Souvik Kundu
| Challenge: | Recent work attributes performance degradation to an exponential decay in hidden-state memory. |
| Approach: | They propose a token filtering strategy that is training-free and attention-guided . they propose 'LAMB' to preserve critical tokens during inference . |
| Outcome: | The proposed token filtering improves long-context performance by 30.35% over state-of-the-art methods on benchmarks. |